A Discussion on Building Practical NLP Leaderboards: The Case of Machine Translation

Recent advances in AI and ML applications have benefited from rapid progress in NLP research. Leaderboards have emerged as a popular mechanism to track and accelerate progress in NLP through competitive model development. While this has increased interest and participation, the over-reliance on single, and accuracy-based metrics have shifted focus from other important metrics that might be equally pertinent to consider in real-world contexts. In this paper, we offer a preliminary discussion of the risks associated with focusing exclusively on accuracy metrics and draw on recent discussions to highlight prescriptive suggestions on how to develop more practical and effective leaderboards that can better reflect the real-world utility of models.Comment: pre-print: comments and suggestions welcom

Credit-worthiness Prediction in Microfinance using Mobile Data: A Spatio-network Approach

Many communities in underdeveloped and developing economies of the world suffer from lack of access to personal credit via formal financial institutions, like banks. However, with the rapid increase in Internet and mobile phone penetration rates, firms are now trying to circumvent this problem using novel technology-enabled approaches. In this research, we leverage a real-world dataset obtained in collaboration with a microfinance firm to show that locational data from mobile phones, coupled with information about communication networks, can be effectively exploited to improve prediction of loan default rates. Specifically, we draw upon recent work in network cohesion based regression modeling to develop a model that uses locational predictors, but within a networked context. We contend that the results from our research can not only illuminate how locational data might be used in assessing creditworthiness, but also empower microfinance firms in resource-poor communities with novel methods for credit scoring

Estimating Policy Effects in a Social Network with Independent Set Sampling

Evaluating the impact of policy interventions on respondents who are embedded in a social network is often challenging due to the presence of network interference within the treatment groups, as well as between treatment and non-treatment groups throughout the network. In this paper, we propose a modeling strategy that combines existing work on stochastic actor-oriented models (SAOM) with a novel network sampling method based on the identification of independent sets. By assigning respondents from an independent set to the treatment, we are able to block any spillover of the treatment and network influence, thereby allowing us to isolate the direct effect of the treatment from the indirect network-induced effects, in the immediate term. As a result, our method allows for the estimation of both the direct as well as the net effect of a chosen policy intervention, in the presence of network effects in the population. We perform a comparative simulation analysis to show that our proposed sampling technique leads to distinct direct and net effects of the policy, as well as significant network effects driven by policy-linked homophily. This study highlights the importance of network sampling techniques in improving policy evaluation studies and has the potential to help researchers and policymakers with better planning, designing, and anticipating policy responses in a networked society